perf(bench): add @cipherstash/bench for index-engagement validation by coderdan · Pull Request #424 · cipherstash/stack

coderdan · 2026-05-05T13:56:13Z

Summary

Adds a new private workspace package @cipherstash/bench that runs encrypted query forms through EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) against a 10k-row fixture and asserts on plan shape — does the integration's emitted SQL engage the canonical EQL functional indexes (eql_v2.hmac_256, eql_v2.bloom_filter, eql_v2.ste_vec), or does it fall back to seq scan?

Surfaces #421 (and informs #422 / #423) on real data instead of unit-test mocks.

Drizzle adapter is wired up; encryptedSupabase and Prisma scaffold to follow.

Layout

packages/bench/
├── sql/schema.sql                                # bench table + 3 functional indexes
├── src/
│   ├── harness/{db,explain,seed}.ts              # pg client, EXPLAIN walker, idempotent 10k-row seed
│   ├── drizzle/{setup,queries}.ts                # Drizzle table, encryptedTable, one builder per operator
│   └── cli/{setup,reset}.ts                      # pnpm db:setup / pnpm db:reset
├── __tests__/
│   ├── db-only.test.ts                           # credential-free smoke (4 tests)
│   ├── harness.test.ts                           # full-stack smoke (uses Encryption client)
│   └── drizzle/operators.explain.test.ts         # plan-shape assertions for #421 / observations for #422
└── __benches__/drizzle/operators.bench.ts        # vitest --bench timings

Layers

EXPLAIN-shape tests (__tests__/) — vitest assertions on plan shape. Cheap, deterministic, intended for CI.
Wall-clock benches (__benches__/) — vitest --bench (tinybench) for median / p95 latency. On-demand.

Run

cd packages/bench
pnpm test -- db-only      # credential-free smoke
pnpm db:setup             # apply schema + seed 10k encrypted rows (requires stash login)
pnpm test                 # full EXPLAIN-shape suite
pnpm bench                # timing benches

Status on this branch

The bench currently fails on eq and inArray — that's the pre-fix repro for the bare-equality bug. The fix is stacked on top in a separate PR.

#422 operators (like / ilike / gt / between / order_by / jsonb_*) all currently seq-scan; the bench captures the plan shape into results/explain-shape.json for the investigation but does not assert.

Test plan

pnpm install at repo root
cd local && docker compose up -d
cd packages/bench && pnpm test -- db-only passes 4/4
pnpm db:setup seeds 10k rows (needs stash login)
pnpm test runs the full EXPLAIN suite (will be partially red until the stacked fix lands)

Example run from #425

changeset-bot · 2026-05-05T13:56:19Z

⚠️ No Changeset found

Latest commit: 0ab1d66

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

coderabbitai · 2026-05-05T13:56:22Z

Warning

Rate limit exceeded

@coderdan has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 30 minutes and 9 seconds before requesting another review.

To continue reviewing without waiting, purchase usage credits in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7668572e-b0ef-4a29-84e3-08dec3bb988d

📥 Commits

Reviewing files that changed from the base of the PR and between 6c67303 and 0ab1d66.

⛔ Files ignored due to path filters (1)

pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml

📒 Files selected for processing (16)

packages/bench/.gitignore
packages/bench/README.md
packages/bench/__benches__/drizzle/operators.bench.ts
packages/bench/__tests__/db-only.test.ts
packages/bench/__tests__/drizzle/operators.explain.test.ts
packages/bench/__tests__/harness.test.ts
packages/bench/package.json
packages/bench/sql/schema.sql
packages/bench/src/cli/reset.ts
packages/bench/src/cli/setup.ts
packages/bench/src/drizzle/setup.ts
packages/bench/src/harness/db.ts
packages/bench/src/harness/explain.ts
packages/bench/src/harness/seed.ts
packages/bench/tsconfig.json
packages/bench/vitest.config.ts

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch dan/bench-package

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Adds a new private workspace package that runs encrypted query forms through EXPLAIN against a 10k-row fixture, asserting that each integration's emitted SQL engages the canonical EQL functional indexes (hmac_256 / bloom_filter / ste_vec) instead of falling back to seq scan. Drizzle adapter is in; encryptedSupabase and Prisma scaffold to follow. Two layers: - __tests__/ — vitest assertions on plan shape (cheap, CI-runnable). - __benches__/ — vitest --bench timings (on-demand). The drizzle/operators.explain.test.ts file currently fails on eq / inArray — that's the pre-fix repro for the bare-equality bug. The fix follows in a stacked branch.

auxesis

@coderdan this is great work expanding test coverage, thank you.

Approved with one comment about incorrect docs.

One other thing to note:

None of these tests will run in in CI when there are changes made to them.
This is by design, because 2/15 tests are currently failing, and that will stop PRs from being merged
I'll put up a follow up PR to add CI, and will rebase it once the fixes are made in the PRs your working on.

coderdan · 2026-05-06T05:05:21Z

None of these tests will run in in CI when there are changes made to them.

This is by design, because 2/15 tests are currently failing, and that will stop PRs from being merged

Yes, that's correct. I plan to enable in CI once the issues have been resolved.

I'll put up a follow up PR to add CI, and will rebase it once the fixes are made in the PRs your working on.

Ok thanks.

coderdan mentioned this pull request May 5, 2026

perf(stack/drizzle): wrap eq/ne/inArray/notInArray in eql_v2.hmac_256(...) #425

Open

3 tasks

coderdan force-pushed the dan/bench-package branch from 44a68f4 to 4f130bb Compare May 6, 2026 02:05

coderdan mentioned this pull request May 6, 2026

ci: run @cipherstash/bench EXPLAIN-shape tests in CI to catch index-engagement regressions #429

Open

coderdan commented May 6, 2026

View reviewed changes

Comment thread packages/bench/src/drizzle/queries.ts Outdated

coderdan commented May 6, 2026

View reviewed changes

Comment thread packages/bench/src/drizzle/setup.ts Outdated

coderdan marked this pull request as ready for review May 6, 2026 03:06

coderdan requested a review from a team as a code owner May 6, 2026 03:06

coderdan force-pushed the dan/bench-package branch from 4f130bb to 0ab1d66 Compare May 6, 2026 03:10

auxesis approved these changes May 6, 2026

View reviewed changes

Comment thread packages/bench/README.md

coderdan merged commit e57b53f into main May 6, 2026
7 checks passed

coderdan deleted the dan/bench-package branch May 6, 2026 05:07

auxesis mentioned this pull request May 6, 2026

ci: run @cipherstash/bench smoke tests on changes #431

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf(bench): add @cipherstash/bench for index-engagement validation#424

perf(bench): add @cipherstash/bench for index-engagement validation#424
coderdan merged 1 commit intomainfrom
dan/bench-package

coderdan commented May 5, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented May 5, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented May 5, 2026 •

edited

Loading

Rate limit exceeded

Uh oh!

Uh oh!

Uh oh!

auxesis left a comment

Uh oh!

Uh oh!

coderdan commented May 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

coderdan commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Layout

Layers

Run

Status on this branch

Test plan

Example run from #425

Uh oh!

changeset-bot Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

coderabbitai Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rate limit exceeded

Uh oh!

Uh oh!

Uh oh!

auxesis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderdan commented May 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderdan commented May 5, 2026 •

edited

Loading

changeset-bot Bot commented May 5, 2026 •

edited

Loading

coderabbitai Bot commented May 5, 2026 •

edited

Loading